Method and system of processing a viewport within large format imagery

ABSTRACT

A computer system for processing a viewport within an image arranged as a matrix of tiles from a container file is disclosed herein. The computer system includes one or more processors configured to: (a) receive a request to read data of the viewport starting from a point of origin of the viewport, the viewport having a pixel width and a pixel height, the viewport being a portion of an image that is stored in a record within the file container; (b) compute column numbers or row numbers, or both, of the tiles containing the viewport; and (c) read in parallel or asynchronously each row or each column of tiles containing the viewport.

CROSS REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. patent application Ser. No.14/273,096 filed on May 8, 2014, now U.S. Pat. No. 9,058,642, which is acontinuation of U.S. patent application Ser. No. 13/963,449, filed onAug. 9, 2013, now U.S. Pat. No. 8,755,609, which is a continuation ofU.S. patent application Ser. No. 13/233,682 filed on Sep. 15, 2011, nowU.S. Pat. No. 8,532,383, which claims the benefit of U.S. PatentApplication No. 61/383,566, filed on Sep. 16, 2010, the entire contentsof which are all incorporated herein by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention pertains to image data management and inparticular to a method and system of processing a viewport within largeformat imagery.

2. Discussion of Related Art

An image generally contains a plurality of pixels (e.g., several hundredmegapixels) and each pixel has one, two or more bands. Each band has acertain color depth or bit depth. For example, an RGB color-based imagehas 3 bands, the red band (R), the green band (G) and the blue band (B).Each of the R, G and B bands can have a depth of 8 bits or more.

An image sensor can be used to capture a series of images, each imagehaving several hundred megapixels. The images may be captured insequence, for example at a reasonably constant frequency. Each image(i.e., each still image) in the sequence or series of images may haveone or more distinct bands and may cover any part of the electromagneticspectrum that can be captured by the image sensor. The image sensor maybe a single sensor or a combination of a matrix of multiple sensorsarranged to generate a single image. Therefore, the series of relativelylarge images captured by an image sensor over a certain period of timeat a near constant capture rate are considered to be large format motionimagery.

BRIEF SUMMARY OF THE INVENTION

An aspect of the present invention is to provide a method of processinga portion of an image. The method includes dividing the image into amatrix of a plurality of tiles, each tile in the plurality of tileshaving a plurality of pixels; defining a viewport within the image, theviewport including one or more tiles in the plurality of tiles, theviewport having an origin point and a width and a height; and reading astrip of pixel data in each column of the matrix of tiles within theviewport starting at a column in the matrix of tiles containing theorigin point. The strip of pixel data starts at a first line of theviewport and ends at the height of the viewport from the first line.

Another aspect of the present invention is to provide a method ofretrieving a viewport within an image arranged as a matrix of tiles froma container file. The method includes reading data of the viewportstarting from a point of origin of the viewport, the viewport having apixel width and a pixel height, the viewport being a portion of an imagestored in a record within the file container; reading record metadata ofthe record; and determining if the record contains an imageuncompressed, tiled arranged in a column major order, or an imagecompressed, tiled and arranged in column major order, or an imagecompressed, tiled and arranged in a row-major order, or in a third partyfile format. The method further includes computing column numbers or rownumbers, or both, of the tiles containing the viewport; and launchingparallel or asynchronous read requests for each row or each column ofthe viewport.

Yet another aspect of the present invention is to provide a method ofretrieving a viewport within an image arranged as a matrix of tiles froma file container. The method includes inputting parameters including anidentification of a record where the image containing the viewport isstored, an origin of the viewport and a width in pixels and a height inpixels of the viewport; requesting the viewport from the file containerusing the parameters; and reading data of the viewport starting from thepoint of origin of the viewport.

Although the various steps of the method are described in the aboveparagraphs as occurring in a certain order, the present application isnot bound by the order in which the various steps occur. In fact, inalternative embodiments, the various steps can be executed in an orderdifferent from the order described above or otherwise herein.

These and other objects, features, and characteristics of the presentinvention, as well as the methods of operation and functions of therelated elements of structure and the combination of parts and economiesof manufacture, will become more apparent upon consideration of thefollowing description and the appended claims with reference to theaccompanying drawings, all of which form a part of this specification,wherein like reference numerals designate corresponding parts in thevarious figures. In one embodiment of the invention, the structuralcomponents illustrated herein are drawn to scale. It is to be expresslyunderstood, however, that the drawings are for the purpose ofillustration and description only and are not intended as a definitionof the limits of the invention. As used in the specification and in theclaims, the singular form of “a”, “an”, and “the” include pluralreferents unless the context clearly dictates otherwise.

BRIEF DESCRIPTION OF THE DRAWINGS

In the accompanying drawings:

FIG. 1 is a schematic representation of a large format motion imageryincluding a series of still images;

FIG. 2 is a schematic representation of a still image in the series ofimages shown in FIG. 1;

FIG. 3A depicts schematically the arrangement of pixels within an image;

FIG. 3B depicts schematically the configuration of a plurality of bandswithin each pixel shown in FIG. 3A;

FIG. 4 depicts schematically an image in which the bytes of all bandsfor a given pixel are arranged for storage together in a row within theimage, according to an embodiment of the present invention;

FIG. 5 depicts schematically an image where all pixels for one band arestored together, followed by all pixels for the next band, according toanother embodiment of the present invention;

FIG. 6 depicts a configuration of tiles within an image, according to anembodiment of the present invention;

FIG. 7 depicts an organization of tiles in a row-major order, accordingto an embodiment of the present invention;

FIG. 8 depicts an organization of tiles in a column-major order,according to another embodiment of the present invention;

FIG. 9 depicts a tiled image with examples of viewports or windowsincluding or covering one or more tiles, according to an embodiment ofthe present invention;

FIG. 10A depicts a tiled image with tiles stored in column-major orderand an example of a viewport within the tiled image, according to anembodiment of the present invention;

FIG. 10B depicts a computational operation for accessing or readingtiles within the viewport from a storage device, according an embodimentof the present invention;

FIG. 11A shows a tiled image having column-overlapped tiles, accordingto an embodiment of the present invention;

FIG. 11B shows an example of overlapped columns in one row of the tiledimage depicted in FIG. 11A;

FIG. 12A shows a tiled image having row-overlapped tiles, according toan embodiment of the present invention;

FIG. 12B shows an example of overlapped rows in one column of the tiledimage depicted in FIG. 12A;

FIG. 13A shows viewports that can be accessed in a tiled image havingcolumn-overlapped tiles using one-seek and one-read operation along theX-axis, according to an embodiment of the present invention;

FIG. 13B shows viewports that can be accessed in a tiled image havingrow-overlapped tiles using one-seek and one-read operation along theY-axis, according to another embodiment of the present invention;

FIG. 14A is a schematic representation of a series or sequence of imagescaptured at a rate of R Hz for a period of time of T seconds, accordingto an embodiment of the present invention;

FIG. 14B is a schematic representation of the series of images stored ina storage device as a plurality or sequence of image files in a foldertree, according to an embodiment of the present invention;

FIG. 15 depicts a logical layout of a container containing a pluralityof image files, according to an embodiment of the present invention;

FIG. 16 depicts a logical layout of the container that holds image filesshowing the positioning of the metadata sections, according to anembodiment of the present invention;

FIG. 17A depicts a layout of a container file having empty recordsreserved within the container file, according to an embodiment of thepresent invention;

FIG. 17B depicts a logical layout of an empty record, according to anembodiment of the present invention;

FIG. 17C depicts a layout of a container file having a series of filerecords extended to contain a series of empty records, according to anembodiment of the present invention;

FIG. 18 is a flow chart of a method for creating a new container orextending an existing container, according to an embodiment of thepresent invention;

FIG. 19 is a flow chart of a method for inserting a very large formatimage within the container, according to an embodiment of the presentinvention;

FIG. 20 is a flow chart depicting a web service for writing or insertinga record into a container file, according to an embodiment of thepresent invention;

FIG. 21 is a logic diagram depicting a system for monitoring andcontrolling insertion of image files into a container, according to anembodiment of the present invention;

FIG. 22 is a diagram showing the hierarchical relationship between aQueue Manager (QM), a container Job Manager (JM), a Insert Service (IS),a Container Creator/Extender Service (CCES) and a Container Record (R);according to an embodiment of the present invention;

FIG. 23 is a diagram showing a single QM running multiple JMs, with eachJM running a job to update a single container C, according to anembodiment of the present invention;

FIG. 24A is a flowchart depicting various operations performed by a JM,according to an embodiment of the present invention;

FIG. 24B is a flow chart depicting various operations performed by eachthread launched by a JM, according to an embodiment of the presentinvention;

FIG. 25 is a flowchart depicting a basic task of a Queue Manager (QM),according to an embodiment of the present invention;

FIG. 26 is a diagram depicting a mathematical procedure used in readoperations to ensure that all read operations always begin at a storagedevice block size (SDBS) aligned offset and are of a size that is alsoSDBS aligned, according to an embodiment of the present invention;

FIG. 27A is a flow chart of a method for retrieval of an image viewportfrom a container, according to an embodiment of the present invention;

FIG. 27B is a flow chart of part of a method for retrieval of an imageviewport from a container, according to an embodiment of the presentinvention;

FIG. 28 is a schematic representation of a viewport within an imagestored within a container file, according to an embodiment of thepresent invention;

FIG. 29 is a flow chart depicting a method for retrieval of one imageviewport from a container, the method being implemented as a webservice, according to an embodiment of the present invention;

FIG. 30A is a flow chart depicting a method for retrieval of multipleimage viewports from a container, the method being implemented as a webservice, according to another embodiment of the present invention;

FIG. 30B is a flow chart depicting a sub-procedure within the methodshown in FIG. 30A;

FIG. 31A is a diagram depicting the internal breakdown of a videosub-component comprising of a video codestream of one or more group ofpictures (GOP), according to an embodiment of the present invention;

FIG. 31B is a diagram depicting a data path of a method for providing astreaming video sub-component as a web service derived from multipleimage viewports from a container, according to an embodiment of thepresent invention;

FIG. 32 is flow chart depicting a method of translating a videosub-component request to multi-record request, according to anembodiment of the present invention;

FIG. 33 is a diagram depicting a method for combining multiple parallelvideo sub-component requests to generate a sequential video codestream,according to an embodiment of the present invention;

FIG. 34A is a flow chart depicting a method of generating a completevideo codestream using multiple parallel video sub-component requests,according to an embodiment of the present invention; and

FIG. 34B is a flow chart depicting a sub-procedure within the methodshown in FIG. 34A.

DETAILED DESCRIPTION OF EMBODIMENTS OF THE INVENTION

FIG. 1 is a schematic representation of a large format motion imageryincluding a series of still images, according to an embodiment of thepresent invention. FIG. 2 is a schematic representation of a still imagein the series of images shown in FIG. 1. As shown in FIG. 1 and FIG. 2,an image “I” captured at time “t” may be “W” pixels wide by “H” pixelstall. FIG. 3A depicts the arrangement of pixels within an image and theconfiguration of the plurality of bands within each pixel, according toan embodiment of the present invention. The pixels are arranged in amatrix layout with a first pixel element P_(0,0) at the upper leftcorner of the matrix and a last pixel element P_(w-1,h-1) at the lowerright corner of the matrix. As shown in FIG. 3B, each pixel P_(i,j)contains b-bands (B₁, B₂, . . . , B_(b)). Each band within each pixelcontains N-bits per band, where N is an integer. In one embodiment, N isgreater than 8 (N≧8) and N is an integer multiple of 8.

The capture rate is approximately “H” Hz. Hence, for a given timeinterval T (e.g., approximately every ½ second), the system or sensormay capture I_(T) images, each image having the aforementioned layoutdepicted in FIG. 2. For example, the image may be 20,000 pixels-wide by20,000 pixels-tall. For example, each pixel may have 3-bands (e.g., RGB)with each band carrying 8-bits.

Generally, pixels in an image are organized as being band-interleaved bypixel (BIP) or band-sequential (BSQ). However, other methods oforganizing pixels also exist. An image is considered to be stored as BIPwhen the bytes for all bands for a given pixel are stored together forany given row within an image. FIG. 4 depicts schematically an image inwhich the bytes of all bands for a given pixel are arranged for storagetogether in a row within the image. For example, as shown in FIG. 4,bytes of bands B₁, B₂, . . . , B_(b) of pixel P_(0,0) are stored in arow followed by bytes of bands B₁, B₂, . . . , B_(b) of pixel P_(1,0),etc. An image is considered to be stored as BSQ, when all pixels for oneband are stored together, followed by all pixels for the next band, andso on. FIG. 5 depicts schematically an image where all pixels for oneband are stored together, followed by all pixels for the next band, andso on. For example, as shown in FIG. 5, pixels having band B₁ are storedtogether followed by pixels having band B₂, etc.

In order to facilitate efficient retrieval of an image, each image inthe image sequence is tiled. An image is broken down into individualimage tiles, each image tile having a specific pixel width and pixelheight. Each image tile is smaller than a single large format motionimage. For example, a tile may be 128, 256, 512 or 1024 pixels wide and128, 256, 512 or 1024 pixels tall for the above mentioned image having20,000×20,000 pixels.

Each image tile may be organized as BIP or as BSQ or using any desiredpixel layout. For example, for BSQ organized images, the raster data foreach band may be tiled separately. FIG. 6 depicts a configuration oftiles within an image. As shown in FIG. 6, the image is broken down intotiles and there are R-rows and C-columns of tiles within the image. Asshown in FIG. 6, each tile is T_(w) pixels wide by T_(h) pixels tall,such that C×T_(w)≧W and R×T_(h)≧H, where W is the width of the image andH is the height of the image.

Generally, tiles for most file formats are organized in row-major orderas an image is often scanned from left-to-right, top-to-bottom. FIG. 7depicts the organization of tiles in a row-major order. In row-majororder, tiles are stored on a storage device such that for each rowR_(j), bytes for tiles for columns 0 to C−1 are stored adjacent to eachother.

However, storing tiles in column-major order may be more efficient forreading and/or updating tiles that are stored without any compression,i.e. uncompressed. FIG. 8 depicts the organization of tiles in acolumn-major order. In column major order, tiles are stored on a storagedevice such that for each column C_(i), bytes for tiles for successiverows 0 to R−1 are stored adjacent to each other, as will be discussedfurther in detail in the following paragraphs. If the image is storeduncompressed, the location of the tiles can be calculated algebraicallyprovided the tile layout order (row or column-major) is known. For tilesthat are stored in compressed format, compression technology,compression format standards, and the implementation of compressiontechnology and standards dictate the efficiency with which image data isstored or retrieved. Hence, in a compressed format, tile storage ordermay become irrelevant. As will be explained further in detail in thefollowing paragraphs, in one embodiment, at least a portion of a tile inthe plurality of stored tiles overlaps another tile in the plurality oftiles (e.g. an adjacent tile).

FIG. 9 depicts a tiled image with examples of viewports or windowsincluding or covering one or more tiles. If each tile is stored as anindependent entity on a storage device, in order to extract, access orread a given viewport such as viewport V₁, V₂ or V₃ within the imageshown in FIG. 9, it may become necessary to read one or more tiles. Inone embodiment, a storage device may be a part of primary storage (suchas main memory) or secondary storage (such as SSD or hard-disk storage)on any general purpose computing platform. A large number ofstandardized file formats support this tiled layout within their storageschema. Examples of image file formats include JPEG, JPEG2000, TIFF, andNITF. Some of these file formats, such as JPEG2000, also store the sizeof each tile as part of the file-header of each tile.

As stated above, one method of storing a single image is by storing theimage as uncompressed tiles in column-major order (as shown in FIG. 8).Another method of storing an image, according to another embodiment ofthe present invention, is illustrated in FIG. 10A. FIG. 10A depicts atiled image with tiles stored in column-major order and an example of aviewport within the tiled image. The viewport has top left corner X₀,Y₀,and V_(w) pixels wide and V_(h) pixels tall. It is assumed that theviewport is wholly contained within the image. In this method, therelative offset for reading or writing a viewport (the viewport havingtop left corner X₀,Y₀, V_(w) pixels wide and V_(h) pixels tall) from animage stored with an uncompressed, tiled scheme in column major ordercan be computed. A write operation, in this case, is an optional read ofthe viewport, an optional modification of existing data in the viewportand a write of the viewport. Offset and size computations for both readand write operations are similar.

As depicted in FIG. 10B, when the order of tile layout is column major,for each column, all H lines are laid out sequentially on the storagedevice. For a given viewport V((X₀,Y₀),V_(w)×V_(h)), instead of havingto read a sequence of tiles that wholly contain the viewport (from thetop left tile T(R₀,C₀) to the bottom right tile T(R₁,C₁)), for eachcolumn (C₀ to C₁), a strip of data is read starting from the first line(Y₀) of that viewport, going down to the height of the viewport (V_(h)lines), as illustrated in FIG. 10B.

As it can be appreciated, a user-defined viewport's dimensions (V_(w)and V_(h)) may not be an exact multiple of T_(w) (tile width) and T_(h)(tile height). Following a column-major order storage schema foruncompressed tiles, in aggregate, the amount of data read from thestorage device can be reduced. For example, in the case shown in FIG.10A, for each column from C₀ to C₁, instead of reading from line numberR₀×T_(h) up to and including line number (R₁×T_(h))−1, the data is readfrom line Y₀ up to and including line number Y₀+V_(h)−1. In oneembodiment, for the top left of the viewport, the position of column C₀is computed as C₀=floor (X₀/Tw) and the position of row R₀ is computedas R₀=floor (Y₀/T_(h)) where T_(w) and T_(h) are respectively, the widthand the height of a tile T. For the bottom right of the viewport, theposition of column C₁ is computed as C₁=floor ((V_(w)+X₀−1)/T_(w)) andthe position of row R₁ is computed as R₁=floor ((V_(h)+Y₀−1))/T_(h)).

In order to read the viewport (e.g., the viewport shown in FIG. 10A),for each column C_(i) that ranges from C₀ to C₁ (including C₁), B_(rw)bytes are read starting from relative offset O_(s)(C_(i)) whereB_(rw)=T_(w)×V_(h)×B_(pp) and O_(s)(C_(i))=((H×C₁)+Y₀)×T_(w)×B_(pp). Theterm relative offset is used deliberately. It is assumed that the imagedata starts at an absolute file offset of O_(s), which may include anoptional header and any preceding data. Thus, the absolute offset forthis location within the image file on a storage device would beO_(s)+O_(s)(C_(i)). Furthermore, in one embodiment, for reading orwriting these tile column strips from C₀ to C₁, each read or writeoperation is dispatched as a parallel asynchronous task. As a result,read operations are parallelized for each column. The part of code thatactually merges data from all columns into a single contiguous viewportmay be also implicitly parallelized.

If each tile T is compressed, the size in bytes S_(T) to be reserved foreach compressed tile T would be S_(T)=S_(TU)/C_(R), where S_(TU) is thesize of each tile in bytes in its uncompressed form and C_(R) is thecompression ratio, where C_(R)>1. The offset O_(s)(T_(k)) is defined asthe number of bytes relative to the start of the image where the datafor tile T_(k) begins in the image. The offset O_(s) is defined as thenumber of bytes from the start of the container file where the data forthe image begins. Tile T_(k) is at column number C_(i) and row numberR_(j) and is represented as T_(k)(C_(i), R_(j)). All tiles are assumedto have substantially a same size S_(T).

Depending on the compression method employed, the actual amount ofcompressed data for a tile may be less than or equal to the size inbytes S_(T) reserved for each compressed tile T. The actual amount ofcompressed data will never be greater than S_(T). Therefore, reservingS_(T) bytes for each tile will wholly contain the compressed data foreach compressed tile of the image. By making the compressed tile size aconstant, the need to store offsets and/or sizes for each tile can beeliminated. As a result, searching through such a list of offsets and/orsizes in order to find the location of a compressed tile can also beeliminated.

Storage devices are generally optimized to read and write data inmultiples of the storage device block size (SDBS). A storage deviceblock size (SDBS) is an integer multiple of the minimum number of bytesthat a storage device is setup to read or write in a single input-output(I/O) operation. This number is generally 512×2″ where n is an integergreater than or equal to 0. Aligning a value to SDBS means increasing ordecreasing the value by an amount such that the resulting value is aninteger multiple of SDBS. For example, if B_(sz) denotes the storagedevice block size SDBS, “A % B” implies the “A modulo B” operation whichis the remainder of division of A by B, and O′ aligns an absolute fileoffset O to SDBS, then O′ can be less than or equal to or greater thanor equal to O. If O′≦O then O′=O−(O % B_(sz)). If O′≧O thenO′=O+(B_(sz)−(O % B_(sz))) % B_(sz).

A storage device can be further optimized to read and write datastarting from a location in the file that is an integer multiple ofSDBS. If S_(T) is not an integer multiple of SDBS, in order to optimizereads and writes, the value of S_(T) can be adjusted to increase tobecome an integer multiple of SDBS. The extra filler bytes can beprovided at the end of the actual data bytes and must be set to nullvalues.

For example, as shown in FIG. 10A, a viewport within a large formatimage can be accessed. The top left tile T₀ containing the top leftpixel (X₀,Y₀) is represented as T₀(C₀,R₀). The bottom right tile T₁containing the bottom right pixel (X₁,Y₁) is represented as T₁(C₁,R₁),where X₁=X₀+V_(w)−1, and Y₁=Y₀+V_(h)−1 (V_(w) and V_(h) beingrespectively the width and height of the viewport).

If the compressed image tiles are stored in column-major order, any tileT_(k)(C_(i), R_(j)) is located at offset O_(s)(T_(k)) such thatO_(s)(T_(k))=S_(T)×(R×C_(i)+R_(j)). If on the other hand the compressedimage tiles are stored in row-major order, any tile T_(k)(C_(i), R_(j))is located at offset O_(s)(T_(k)) such thatO_(s)(T_(k))=S_(T)×(C×R_(j)+C_(i)). Hence, the location of eachcompressed tile containing the requested viewport within the image canbe computed rather than looked up.

If a display device that is configured to display a viewport from theimage has an upper limit to its viewport width and height, a method canbe used to minimize read time by creating tiles containing pixels thatoverlap pixels from adjacent tiles. For example, the display device hasa pixel width D_(w) and pixel height D_(h). The display device can beconfigured to display at least a portion of an image having pixel widthW and pixel height H. The image can be stored in the tiled formatspecified above such that 0<D_(w)<T_(w) and 0<D_(h)<T_(h).

FIG. 11A shows a tiled image having column-overlapped tiles. FIG. 11Bshows an example of overlapped columns in one row of the tiled imagedepicted in FIG. 11A. FIG. 12A shows a tiled image having row-overlappedtiles. FIG. 12B shows an example of overlapped rows in one column of thetiled image depicted in FIG. 12A.

In one embodiment, a display device or processing device having aspecified upper limit to its viewport pixel width D_(w) and pixel heightD_(h) can be provided to display or process a viewport from the image ofwidth W and height H pixels. For example, the image can be stored in thetiled format specified above such that 0<D_(w)<T_(w) and 0<D_(h)<T_(h).A method can be used to minimize read time to one seek and one readoperation by creating tiles containing pixels that overlap pixels fromadjacent tiles.

An overlap of P_(c) pixels provides that two adjacent columns haveP_(c)×T_(h) area that has the same pixels in both tiles. Similarly, anoverlap of P_(r) pixels provides that two adjacent rows have P_(r)×T_(w)area that has the same pixels in both tiles. The displayed image portioncan be overlapping either along a column or along a row, or both. Theimage data can be stored in this overlapping manner on a storage deviceso that redundant data can be generated to facilitate efficient read fordisplaying on a display device.

If a number of pixels of overlap P_(c) is equal to 0, no overlap ispresent between adjacent tiles. If D_(w)≦P_(c) (where P_(c) is theamount of column overlap in pixels), then one and only one column willcontain the entire width of the requested viewport. When the displayedwidth D_(w)≦P_(c) (where P_(c) is the amount of column overlap inpixels) and the displayed height D_(h)≦P_(r) (where P_(r) is the amountof row overlap in pixels) any desired viewport within the image may bedisplayed using a single seek and read operation from the storagedevice. As it can be appreciated, the image can be overlapped eitheralong a column, along a row, or both.

For example, in the case where each tile is stored uncompressed(assuming that ample storage is available), if the image is storeduncompressed on a storage device following a column-major order asdescribed in the above paragraphs, the image does not require rowoverlapping to fulfill the above single seek and read requirement. Rowoverlap is implicit. This is because within a column, one can accessdata from any line to any line. It would therefore only need columnoverlapping.

On the other hand when each tile is stored compressed (assuming thatample storage is available) and a single seek and read operation off thestorage device is desired, line-by-line access is no longer possiblebecause compressed tiles are whole tiles, encoded as a single entity.Therefore, in order to access the displayed viewport, both row andcolumn overlapping that has D_(w)≦P_(c) and D_(h)≦P_(r) would be needed.This would involve the application of both the methods depictedschematically in FIGS. 11A, 11B, 12A and 12B.

If the above constraints for overlapping are implemented, a viewport ofpixel width D_(w) and pixel height D_(h) is always contained within oneand only one tile T_(v)(C_(v), R_(v)). In other words, an area orviewport D_(w)×D_(h) displayed on a displayed device is contained withina single tile T.

FIG. 13A shows viewports that can be accessed in a tiled image havingcolumn overlapped tiles using one-seek and one-read operation along theX-axis, according to an embodiment of the present invention. Forexample, tile T₀(C₀, R₀) contains the viewport's top left corner suchthat C₀=2×(X₀/T_(w)) and R₀=2×(Y₀/T_(h)), and tile T₁(C₁, R₁) containsthe viewport's bottom right corner such that C₁=2×(X₁/T_(w)) andR₁=2×(Y₁/T_(h)), where X₁=X₀+D_(w)−1, and Y₁=Y₀+D_(h)−1 and D_(w) andD_(h) are, respectively, viewport pixel width and viewport pixel heightD_(h).

Based on these formulae, |C₁−C₀| may be equal to 0 if X₀ is in the lefthalf of the tile (as shown in FIG. 13A, viewport V₁ in both parts) ormay be equal to 2 if X₀ is in the right half of the tile (as shown inFIG. 13A, viewport V₂ in both parts). If C₁=C₀ then C_(v)=C₀ otherwiseC_(v)=C₀+1. By applying the formula in FIG. 10B to the overlapped layoutin FIG. 13A, X₁ would be X₀+D_(w)−1. There are 5 possible combinationsof where X₀ and X₁ could be, as depicted in FIG. 13A. (1) Both X₀ and X₁fall only in column C. (2) X₀ falls only in column C, X₁ falls in columnC and C+1. (3) X₀ falls in column C and C+1, X₁ falls in column C andC+1. (4) X₀ falls in column C and C+1, X₁ falls only in column C+1. (5)Both X₀ and X₁ fall only in column C+1. In FIG. 13A, Viewport V₁ coverscases (1) and (2). In FIG. 13A, Viewport V₂ covers cases (4) and (5).When applying the formula in FIG. 10B, |C₁−C₀| results in a value of 0for cases (1) and (2), because C₁ evaluates to C, and C₀, evaluates toC, thus implying that X₀ and X₁ are both in column C. Thus C_(v)=C₀. Forcases (4) and (5), |C₁−C₀| results in a non-zero value, implying that X₀and X₁ are both in column C+1. Thus, C_(v)=C₀+1. In case (3), since bothare in C and C+1, |C₁−C₀| results in a zero value implying that X₀ andX₁ are both in column C. Thus C_(v)=C₀.

FIG. 13B shows viewports that can be accessed in a tiled image havingrow-overlapped tiles using one seek and one read operation along theY-axis, according to another embodiment of the present invention.Similarly, based on the above formulae, |R₁−R₀| may be 0 if Y₀ is in theupper half of the tile or may be 2 if Y₀ is in the lower half of thetile. If R₁=R₀ then R_(v)=R₀ otherwise R_(v)=R₀+1.

Very large format motion imagery generally requires a significant amountof storage per frame. When using multiple sensors, the amount of storageper frame can increase linearly. Also, with an increase in bit-depth tomore than 8 bits per pixel, the amount of storage per frame can alsoincrease linearly. In addition, with an increase in number of bands(from luminance to RGB or RGB+Near IR), the storage amount per frame canalso increase linearly. Furthermore, as sensor size (width W and heightH) increases, the amount of storage per frame can increaseexponentially.

For instance, an existing sensor generates very large format motionimagery at 150 megapixels (MPixels) per image. The sensor may capturethis data at about 2 Hz, i.e., two images per second. The image data iscaptured as 1-band (luminance) at 8-bits/band. Uncompressed, each imageis about 150 megabytes (MB). When the image is compressed, using forexample the JPEG2000 codec at about 8:1, the size of the image data canbe reduced to about 19 MB per image. For each hour, this amounts toabout 7200 images occupying 1 terabyte (TB) of uncompressed data or 135gigabytes (GB) of compressed data. If this data were to be collected for10 hours each day, this amounts to capturing about 26 million images peryear, totaling about 3.6 petabytes (PB) uncompressed or about 500 TBcompressed at the above compression ratio.

As another example, if a 1000 MPixel (one gigapixel) image sensorcaptures 16-bits per band, 3-bands at 2 frames per second, the amount ofdata captured for 10 hours per day, for one year would be about 140 PBof uncompressed images or 17.5 PB compressed images at the abovecompression ratio. At night, an IR sensor at 1000 MPixels at 16-bits perband, 1-band at collecting at 2 frames per second, compressed at theabove ratio, would be an additional 26 million files and an additional 6PB.

As yet another example, if there are 200 sensors, each at 1000 MPixels,collecting data daily, then over 5 billion individual compressed filescan be collected, massing to about 3500 PB of compressed images at theabove compression ratio for the day time and an additional 5 billionfiles, at 1100 PB for the night time collections.

Dealing with such a large number of files can be a performance andscalability challenge for existing file systems. Using standard existingfunctions and tools to read or write such files is an extremelyinefficient process with conventional technologies.

In the following paragraphs, a method of storage of large format imageswithin a single large file on a storage device is described. The methodallows for efficient update and retrieval of any viewport from anyimage. The single large file is considered to be a container for a largenumber of substantially homogenous very large format image files. Theimage files are stored into a single file. The term “homogenous files”is used herein to mean the image files are all logically similar, i.e.,the image files are obtained from a sensor that delivers images ofsubstantially the same pixel width and height (i.e., substantially samesize) at substantially the same band count and bit depth. Furthermore,in homogenous image files, the compression ratio of each of the imagefiles is also similar.

FIG. 14A is a schematic representation of a series or sequence of I_(T)images captured at a rate of R Hz for a period of time of T seconds,according to an embodiment of the present invention. FIG. 14B is aschematic representation of the series of images stored in a storagedevice as a plurality or sequence of image files in a folder tree,according to an embodiment of the present invention. A “container” isherein defined as a single large file that contains a plurality orsequence of homogenous image files. Once the image files are storedwithin the container, they are deemed “containerized.”

FIG. 15 depicts a logical layout of the container, according to anembodiment of the present invention. The container contains one filecontainer metadata section per container which includes informationabout the container such as what the container contains, how theinformation is organized, etc. The file container further contains aplurality of records. For each image file in the sequence of imagefiles, there is one corresponding record in the container. Each recordcomprises record metadata, image metadata, and the image file. Therecord metadata includes information about a specific record within thecontainer such as the format of the image file, layout of tiles (ifany), list of relative tile position and size, compression state (i.e.,compressed or not compressed), etc. The image metadata includesinformation about the image such as acquisition time, creation time,modification time, security parameters, geospatial information, notes,etc. The image file comprises the actual image data that is beingcontainerized (i.e., images I₁, I₂, . . . , I_(T)). The container isstored on a known file system. In one embodiment, the container may be afile having a plurality of “fixed length records.” A record includesrecord metadata, image metadata, and the image file itself. The term“fixed length record” file means the sizes of the space reserved for the“record metadata,” the “image metadata,” and the “image file” is thesame for all records. If the image data is compressed in the image file,the size reserved in a record to hold its corresponding image fileequals the size of the largest file in the collection of input files. Ifthe size of the largest file in the collection of image files is notknown, a value can be computed by taking the uncompressed image size anddividing it by the compression ratio (C_(r) where C_(r)>1) that iscommon to all images. If C_(r) is not common to all images, then theminimum value of C_(r) for all images is used in the computation.

FIG. 15 depicts a logical layout of a container that holds compressedimage files, according to an embodiment of the present invention. Thecontainer contains one file container metadata, and a plurality ofrecords. Each record comprises record metadata, image metadata and thevery large format image file data. The file container metadata, therecord metadata, the image metadata and each image file are stored on astorage device starting from an absolute file offset that is aligned toSDBS. The amount of bytes reserved to hold the file container metadata,the record metadata, the image metadata and each image file is aninteger multiple of the SDBS.

A storage device block size (SDBS) is an integer multiple of the minimumnumber of bytes that a storage device is setup to read or write in asingle I/O operation. In one embodiment, this number can be 512×2^(n)where n is an integer≧0. Aligning a value to SDBS means increasing ordecreasing the value by an amount such that the resulting value is aninteger multiple of SDBS. For example, if B_(sz) denotes SDBS, “A % B”represents the “A modulo B” operation which is the remainder of divisionof A by B, and O′ aligns an absolute file offset O to SDBS, then O′ caneither be ≦ or ≧O. If O′≦O then O′=O−(O % B_(sz)). If O′≧O thenO′=O+(B_(sz)−(O % B_(sz))) % B_(sz).

The file container metadata section is of a known fixed size. The sizeis preset by a user of the system and depends on the nature of over-allmetadata to be stored in the file. The file container metadata is storedin the file from the start of the file. It is stored such that the filecontainer metadata section is contained within an integer multiple ofSDBS. A software application for accessing the container can beconfigured to interpret the file metadata section. In one embodiment,the size of the file container metadata section is stored as part of theinformation of the file container metadata section. The first SDBS bytesof the section contain, amongst other values, the status flags for theentire container file as well as the value of SDBS itself. The firstSDBS bytes of the section also contain additional data that specifyrelative starting point and the size of the “record metadata” section,the “image metadata” section, and the amount of bytes reserved for theimage file itself. The file container metadata section also holdsinformation regarding the common format of the files. The informationregarding common format may include information such as all files may beuncompressed, compressed using the JPEG2000 codec, or the JPEG codec, ortiled with each tile compressed using run-length encoding (RLE)technique, etc. The file container metadata section also holds dates,owner information, and security features for the file container and anyadditional metadata elements that individuals that processed the filecontainer may have generated. The file container metadata section mayfurther hold the total record count. For example, a predefined datastructure is employed to hold this metadata. The industry wide generalmodel of storing metadata as key-length-value sets may be followed forinformation custom to the type of data stored in the file. Thepre-reserved space for the file container metadata section may alsoinclude extra space just in case a user of the file containeranticipates additional metadata to be generated at a later date.

A sequence of records is stored following the file container metadatasection. Each record holds a single very large format image file. Arecord is divided into three primary sections: the record metadatasection, the image metadata section and the image file section. Therecord metadata section is the first section within the record. Theimage metadata section may precede or follow the image file section.Relative offsets of the image metadata section and the image filesection are stored within the record metadata section.

The record metadata section stores information specific to thecorresponding record. The record metadata has a fixed size in bytes forall records. The size of the record metadata may be known from the filecontainer metadata section. In one embodiment, the record metadata isstored in the file container such that each record metadata sectionstarts from a location that is aligned to an integer multiple of thestorage device block size (SDBS). The number of bytes in the recordmetadata section is also contained within an integer multiple of SDBS.The record metadata section contains record-specific metadata. In oneembodiment, the first SDBS bytes of the record hold, amongst othervalues, status flags. In one embodiment, the record metadata holdsstatus flags of associated record (i.e., valid, empty, invalid, or beingupdated and how the record is being updated). If a corresponding imagefile of the record is tiled and compressed, the record metadata sectionfurther holds data about the compression codec, relative offsets andsizes of each tile in the associated image file. In one embodiment, therecord metadata section can also hold record creation and modificationtimes, user, computer and application names of who modified the record,etc. In one embodiment, additional housekeeping data may also be storedin the record metadata section.

The image metadata section holds information specific to the associatedimage. For instance, the image metadata section may include the UTC timeof acquisition (e.g., in microseconds), geospatial information about theimage, sensor parameters, parameters from the capturing platform ordevice, parameters on the environment in which the image was captured bythe device (e.g. airborne capture device elevation, pitch, yaw androll), original creating and modification times of the image file, theoriginal file name, the folder in which the file existed, etc. In oneembodiment, additional housekeeping data may also be stored in the imagemetadata section. In addition, in one embodiment, storage space can bereserved per image to allow a user to add notes, etc. In one embodiment,if the image file stored in the image file section does not containR-sets, then R-sets may be computed and stored in the image metadatasection. An R-set is also known as a reduced resolution dataset or RRD.An R-set is a single scaled-down version of the original image. MultipleR-sets can be stored per record, each being subsequently smaller thanthe previous. Each R-set is an image and follows the same rules forstorage as the large format image stored in the image file section.

The image file section contains the very large format image file itself.If the image is stored as an uncompressed raster, the image is tiled ina form described above. If the image is stored as a compressed raster,each tile is compressed and the image as a whole is compressed in a formdescribed above. If the image is stored as a file of a known format, theimage file is analyzed and information about the image file may bestored in the appropriate metadata sections, as discussed in the aboveparagraphs. By the time a record is created from the source data of avery large format image, all information needed to efficiently extractor decode a viewport from the very large format image is extracted orgenerated and stored in the record or image metadata sections. The imagefile section begins at an SDBS aligned offset and is stored in a sizethat is SDBS aligned.

FIG. 16 depicts a logical layout of a container file that holds imagefiles showing the positioning of the metadata sections, according to anembodiment of the present invention. As depicted in FIG. 16, the filecontainer metadata section is stored at the header of the file containerand is SDBS aligned. The record metadata sections is stored adjacent tothe file container metadata is also SDBS aligned. In one embodiment, theimage metadata section is stored adjacent the record metadata sectionand is SDBS aligned. Following the image metadata section is stored theimage file itself. The image file itself is also SDBS aligned. However,the image metadata section can also be stored following the image file.

FIG. 17A depicts a layout of a container file having empty recordsreserved within the container file, according to an embodiment of thepresent invention. As depicted in FIG. 17A, the file container metadatasection is written at the header of the file container followed by thereserved space for the empty records.

FIG. 17B depicts a layout of an empty record, according to an embodimentof the present invention. As shown in FIG. 17B, the empty recordincludes a record metadata section where the record metadata can bestored. The record metadata section is marked empty when there is nometadata stored therein. The empty record further includes imagemetadata section reserved for storing image metadata. As shown in FIG.17B, the image metadata section is marked space reserved no valid dataindicating that there are no image metadata stored in the image metadatasection. The empty record also includes an image file section. The imagefile section is configured to store the image file itself. As shown inFIG. 17B, the image file section is marked “space reserved with no validdata” as there are no image yet stored in the image file section of therecord, thus the record labeled as empty.

FIG. 17C depicts a layout of a container file having a series of filerecords and extended to contain a series of empty records, according toan embodiment of the present invention. In order to extend records in afile container, i.e., to add record spaces within the file container,the number of records by which to extend the dataset can be specified.For example, a user can instruct the system using an extension procedureto add records from the end of the container file to create an extensionof the container file. At the end of the extension procedure, the filecontainer metadata section is updated to reflect the new count ofrecords. In this case, while the container is being extended, the filecontainer marks the status of the file as being extended. When acontainer is being created or extended, it can be opened by any otherprocess for read-only operations.

A new container file can be created or an already created container filecan be extended as desired using a method referred to herein ascontainer creator or extender (CCE), according to an embodiment of thepresent invention. In one embodiment, at any point in time, only one CCEcan update a container file. In one embodiment, the CCE method can beimplemented programmatically as will be described in detail in thefollowing paragraphs.

In one embodiment, when creating a new container file, the CCE methodperforms three steps. The first step to creating such a large containerfile is identifying the data being laid into the container file. Thiscan be performed with inputs from the user. Some of these inputs may bethe pixel width, height, band-count, and band data type of each image,the source data organization, the number of images that the useranticipates to be stored into the container file, the maximum size perrecord of custom user-defined metadata that the user wishes to store inthe image metadata section, additional image metadata (if alreadyavailable) and other parameters. Source data organization refers to theway in which source data is going to be stored in the container file. Asingle image can be stored as a sequence of compressed or uncompressedimage tiles that combine to form the very large format image. The imagecan also be stored as a source image file of a known file format suchthat most of the source image files are nearly the same size in bytes ona storage device. Using this information, one can compute the size ofall three sections within a record.

The second step is to create an empty record, where the information thatthe user entered in the first step is the only valid information storedwithin the record. The size of the image metadata and image filesections is computed and set to align itself to the SDBS.

The final and third step creates the container file, and writes the filecontainer metadata. The status of the file container metadata is markedas being “updated.” The third step then writes empty records into thefile, sequentially from the first to the last anticipated record.

An empty record is written to the container by taking an empty recordcreated in the second step, marking the status of the record as being“updated” and writing the entire record into the container file. Aftercompleting writing the record, the status of the record is marked as“empty” and the first SDBS bytes of the record are rewritten.

After the container is created, the CCE method resets the status of thefile container metadata as “available,” and writes the first SDBS bytesof the file container metadata to the file and closes the filecontainer.

In one embodiment, when extending an existing container file, the CCEmethod also performs three steps. The size of a record and itssub-sections is determined from the file container metadata section. Thenumber of records to add is known from user input. The empty record canthen be created from having identified the data. The container isopened. The status of the file container metadata is marked as “beingupdated.” The desired number of empty records is appended to thecontainer file, one record at a time, following the same rules asdescribed in the above paragraphs. In one embodiment, the containermetadata section is updated to reflect a new count of records. After thecontainer has been created, the CCE method resets the status of the filecontainer metadata as “available,” writes the first SDBS bytes of thefile container metadata to the file and closes the container.

While an empty container is being created or an existing container isbeing extended, it can be opened by any process other than a CCE, forread or write operations.

In one embodiment, the CCE method can be implemented as a web service,i.e. a CCE service (CCES). As an example, a client or user can use anHTTP URL string as an interface to the CCES. Information exchangebetween the service and a client of the service can be implemented as anindustry standard interface, such as XML. Since a CCES may take time toperform its tasks, a CCES reports its progress during the time it isrunning, and do so regularly, for example, at least once every fewseconds, to the requesting client.

FIG. 18 is a flow chart of a method for creating a new container orextending an existing container, according to an embodiment of thepresent invention. For example, the method can be implemented as a webservice (a CCE service or CCES). The CCE method can be initiated by auser requesting for a CCE operation on a container, at S10. The CCEmethod includes validating inputs from a user, at S12. The inputs fromthe user include various parameters of the container including thenumber of records, size of each record, etc., as described in the aboveparagraphs. The CCE method further includes inquiring whether thedesired operation is for creating a container or extending a container,at S14. If the desired operation is for creating a container, the methodvalidates user inputs, at S16. The method then inquires whether acontainer already exists, at S18. If the container does not exist, themethod creates a new container, at S20. If the container already exits,the method reports the error to the user, at S22. At any phase before orafter creating a new container, the method can update the user withstatus information, at S24. If the desired operation is for extending acontainer, the method validates user inputs (e.g., various parameterssuch as number of records, size of each record, etc.), at S26. Themethod then inquires whether a container already exists, at S28. If thecontainer already exits, the method opens the existing container, atS30. If the container does not exist, the method reports an error, atS22. Similarly, at any phase before or after opening an existingcontainer, the method can update the user with status information, atS24. In both cases of creating a new container or opening an existingcontainer to extend the existing container, the method writes or storesa record into the container, at S32. The method repeats this operationuntil all desired number of records are written into the container byinquiring whether desired records are written into the container, atS34. The method then reports timing results and success or failure inwriting the records into the container to the user, at S36 and ends atS38.

FIG. 19 is a flow chart of a method for inserting a very large formatimage within the container, according to an embodiment of the presentinvention. The method for inserting a single record includes inputting aname of the container file and record number to insert into, inputting avery large format image as a raw image or in a known file format, andany additional parameters. The method for inserting or writing a singlerecord includes reading and analyzing source data, at S40. The methodincludes opening a container where the image will be stored, at S42. Themethod further includes inquiring whether the source data is valid forbeing entered into the record, at S44. If there are any problems withthe source data (i.e., data source is not valid), the method proceeds toreporting an error, at S46. Otherwise, if the data source is valid, themethod proceeds by preparing the record and image metadata, at S48.After preparing the record and image metadata, the method furtherproceeds by preparing the record itself, at S50. The method theninquires whether the record exists, at S52. In one embodiment, theinquiry includes determining whether F_(sz)>C_(sz)+R_(sz) (N+1), whereF_(sz) is the current container file size, C_(sz) is the file containermetadata size, and R_(sz) is the record size, and N is the record numberbeing updated. If the record does not exit, the method reports an errorat S46. If the record exits, the method further inquires whether thefile container metadata status is set to “available”, at S54. If thefile container metadata status is set to “available” at S54, the methodfurther inquires whether the record is the last record, at S56. If themethod returns that the record is the last record, the method reports anerror at S46. Otherwise, if the record is not the last record, themethod writes the entire record with the status “being updated”, at S58.Similarly, if the file container metadata status is set to available, atS54, the method writes the entire record with the status “being updated”at S58. After writing the entire record with status “being updated” atS58, the method marks the record as either being “valid” at S60 orinvalid (if there are any problems while writing the record) at S62.After writing the entire record, the method proceed to rewriting SDBSbytes of record, at S64. If the writing of the record and the rewritingof the storage device block size (SDBS) is successful, the methodproceeds to close the container, at S66 and report that the writing ofthe record and the SDBS bytes of the record are successful, at S68 whichends the method, at S69.

In one embodiment, this method can be implemented as an insert service(IS). For example, a HTTP URL string can be used as an interface toimplement the insert service (IS). FIG. 20 is a flow chart depicting aninsert service (IS) for writing or inserting a record into a containerfile, according to an embodiment of the present invention. In oneembodiment, the service can be implemented using a computer system, forexample a networked computer system having a world wide web connection.Information exchange between the service and a user of the service is anindustry standard interface, such as XML. The service or method isinitiated by a user requesting to write a record into a container byinputting a source file name, container name, record number and otherparameters, for example using HTTP. The insert service receives a HTTPrequest with source file name, container name, record number and otherparameters, at S70. Upon entry of the various parameters includingsource file name, container name, etc., the service verifies thevalidity of the request, at S72 and acknowledges the request and sendsupdate information to the user, at S73. The service or method thenproceeds to read and analyze the source data provided by the user, atS74. The service or method then proceeds to open the container where therecord or source data will be stored, at S76. The method furtherincludes determining whether the source data is valid to be entered intoa record within the opened container, at S78. If the data source is notvalid, the method proceeds to reporting an error at S79. If the datasource is valid, the method proceeds by preparing the record and imagemetadata, at S80. After preparing the record and image metadata, themethod further proceeds by preparing the record, at S82. The method theninquires whether the record exists, at S84. In one embodiment, theinquiry includes determining whether F_(sz)>C_(sz)+R_(sz) (N+1), whereF_(sz) is the current container file size, C_(sz) is the file containermetadata size, and R_(sz) is the record size, and N is the record numberbeing updated. If the record does not exist, the method reports an errorat S79. If the record does exit, the method further inquires whether thefile container metadata status is set to “available” at S86. If the filecontainer metadata status is set to “available” at S86, the methodfurther inquires whether the record is the last record, at S88. If themethod returns that the record is the last record, the method reports anerror at S79. Otherwise, if the record is not the last record, themethod writes the entire record with the status “being updated”, at S90.Similarly, if the file container metadata status is set to available, atS86, the method writes the entire record with the status “being updated”at S90. After writing the entire record with status “being updated” atS90, the method marks the record as either being “valid” at S92 orinvalid (if there are any problems while writing the record) at S94.After writing the entire record, the method proceeds to rewriting SDBSbytes of record with a valid status, at S96. If the writing of therecord and the rewriting of the storage device block size (SDBS) issuccessful, the method proceeds to close the container, at S97 andreport that the writing of the record and the SDBS bytes of the recordare successful, at S98 which ends the method, at S99.

A method for inserting a sequence of very large format images within acontainer is described in the following paragraphs. In one embodiment,the method for inserting a sequence of images into a container includesusing multiple computing platforms running a multi-threaded “insertsequence” tool such that each thread inserts one very large format imageinto one record and no two threads for a process and no two processesacross multiple computing platforms update the same record in the samecontainer simultaneously. In the context of this application,multi-threaded means a process that launches and executes one or morethreads.

The method for inserting a single image within a container, as describedabove, is a fairly independent and isolated process. Hence, it would notmatter if at any given point in time, several very large format imageswere being inserted into multiple records simultaneously, as long as notwo very large format images are assigned to a same record number. Inone embodiment, each insert can be executed as a service. Therefore,parallel processing across multiple computers can be implemented toinsert multiple image files into record spaces of a containersubstantially simultaneously. In one embodiment, the parallel processingof inserting images can be monitored and controlled with higher-leveltools.

FIG. 21 is a logic diagram depicting a system for monitoring andcontrolling insertion of image files into a container, according to anembodiment of the present invention. As shown in FIG. 21, thehigher-level system for monitoring and controlling the insertion ofimage files into a container may be provided with a three levelhierarchy. In one embodiment, a first level consists of a singleoperational instance of a Queue Manager (QM) that oversees thesubmission of very large format images to specific containers on astorage device. The first level is a common location where all jobs aremanaged. A “job” consists of the task of inserting one or more verylarge format images into a container at known record locations orspaces. In one embodiment, there may be multiple QMs running acrossmultiple interconnected computing platforms. One QM is the main managerwhile the remaining other QMs are failovers.

In one embodiment, a second level in the monitoring system consists of asingle operational instance of a container Job Manager (JM_(f)). Asingle operational instance of a container JM_(f) is assigned to asingle container file C_(f). In one embodiment, the container JM_(f)retrieves the next job of inserting one or more images into a containerfrom QM for the C_(f) that is associated with the container JM_(f). Inone embodiment, the container JM_(f) launches multiple, simultaneousHTTP requests to a pool of available insert services IS with each ISgetting a unique record to update within the C_(f). By monitoring theresponse from an insert service IS, the container JM_(f) can monitor thestatus of each request and therefore perform any error management orrecovery. JM_(f) cannot launch simultaneous IS requests to the samerecord number. JM_(f) cannot launch simultaneous CCES requests. At theend of each job, a container JM_(f) updates the file container metadatasection for its associated container file C_(f) if needed. In oneembodiment, at any instance, the JM_(f) does not launch parallelrequests to the same record space (i.e., a record with a same IDnumber).

In one embodiment, a third level consists of several independent InsertServices (IS). Each IS performs the method or procedure depicted inFIGS. 19 and 20 and described in detail in the above paragraphs andreports incremental progress to its parent JM_(f) via a serviceinterface of the insert service. In one embodiment, IS parameters mayalso be setup such that if a request comes in to update only the imagemetadata or part of the metadata, it follows the same logic as shown inFIGS. 19 and 20. However, instead of updating the entire record, therequest updates only the record metadata and image metadata portion ofthe record. As described in the above paragraphs, a CCES either extendsan existing C_(f) or creates a new C_(f) if so requested by the user.The CCES reports incremental progress to its parent JM_(f) via itsservice interface.

FIG. 22 is a diagram showing the hierarchical relationship between theQueue Manager (QM), the container Job Manager (JM), the Insert Service(IS) or Container Creator/Extender Service (CCES) and the ContainerRecord (R). As shown, since the JM for a container manages implicitsynchronization for each job and the IS manages updating each record,the actual process of updating records can be made to go in parallelspread across one or more instances on one or more computing platforms.The hierarchy of individual modules QM, JM, IS/CCES and R also allowsfor a single integrated tool such that all three levels or tiers can bein the same application. For example, in one embodiment, an insertservice (IS), instead of being distributed as multiple instances acrossmultiple computers, would be one or more threads within a singleapplication.

FIG. 23 is a diagram showing a single Queue Manager QM running multiplejob managers JMs, with each JM running a job to update a single distinctcontainer C, according to an embodiment of the present invention. Asshown in FIG. 23, a JM is associated to a specific container C₁, C₂, . .. , or C_(N). The JMs running jobs on the containers can be run inparallel. In one embodiment, no two JMs running in parallel can updatethe same container at the same time. Each job Manager JM_(i) gets itsown container C_(i).

The QM launches a JM associated with a container, monitors the JM andprocesses messages supplied by the JM. The JM validates the job,prepares a list of CCES or IS requests that need to be processed tocomplete this job, launches processor threads waits for the processorthreads to end while periodically reporting the status of the JM to theuser.

FIG. 24A is a flowchart depicting various operations performed by a JM,according to an embodiment of the present invention. As shown in FIG.24A, the JM inquires whether a job from the QM is valid for itsassociated container C, at S100. If the job from the QM is not valid forthe container C associated with the JM, the JM reports an error to theQM, at S102. If the job from the QM is valid for the container Cassociated with the JM, the JM launches a finite number of threads, atS104. A thread can either be a CCES or an IS dispatcher which creates,extends a container or inserts a record into a container, as isdescribed further in detail in the following paragraphs. The JM thenprepares a task list of CCES and IS operations to perform on containerC, at S106. The JM then inquires whether all tasks have been processed,at S108. If all tasks have been processed, the JM sends a completionmessage to the QM, logs and terminates, at S110. If the tasks have notall been processed, the JM interprets and processes status messages fromall threads and summarize, at S112. The JM sends a status update messageto the requesting QM periodically (e.g., every few seconds), at S114.

FIG. 24B is a flow chart depicting various operations performed by eachthread launched by a JM, according to an embodiment of the presentinvention. Each thread includes getting a next task to perform from theJM task list, at S120. If no tasks are remaining from the task list, thethread is ended, at S122. If, on the other hand, a task is present, thethread inquires whether the task is a CCES task or an IS task, at S124.If the task is a CCES task, the thread prepares and sends a CCES HTTPrequest, at S126. The CCES procedure takes time to run, therefore theresults are streamed periodically (e.g., as a multipart reply). Thethread also includes keeping a status message box up-to-date. The threadfurther includes inquiring whether the CCES request completed beforetimeout, at S130. If the CCES request did not result in a valid responsebut timed out instead, this request is marked as failed and a failurereport is generated at S131. A report is send to S138. If the CCESrequest is completed before timeout, the thread receives success/failurereport from the request and update status box, at S137. The thread theninquires whether the request is successful, at S138. If the request issuccessful, the thread informs the JM about the success of the request,informs the JM about timing and report if there is a partial failure andother findings, at S140. If the request is not successful, the threadlogs the failure and updates status message box, at S142.

If the task is an IS task, the thread prepares and sends an IS HTTPrequest, at S125. The thread then receives acknowledgement from the ISand retain the IS request, at S127. The thread further includesinquiring whether the IS request completed before timeout, at S129. Ifthe IS request did not complete before timeout, the thread inquireswhether the IS request was resent N times, at S132. The number N is aninteger greater than or equal to 1 and is a user configured value. Inone embodiment, if the IS request was not sent N times, the threadoptionally waits a known time and resends the request, at S134. If theIS request was sent N times and still each request timed out, the threadmarks the request as failed and generates a failure report at S136. Ifthe IS request is completed before timeout, the thread receivessuccess/failure report from the request and updates status box, at S137.The thread then inquires whether the request is successful, at S138. Ifthe request is successful, the thread informs the JM about the successof the request, informs the JM about timing and report if there is apartial failure and other findings, at S140. If the request is notsuccessful, the thread logs the failure and updates status message box,at S142.

FIG. 25 is a flowchart depicting the basic task of a Queue Manager (QM),according to an embodiment of the present invention. A user posts jobson a queue and monitors the jobs. A QM initiates or starts JMs forcontainers that the QM is assigned. The QM retrieves a pending job fromits job queue and marks the job as being “in-processing”, at S150. TheQM inquires whether the container (output container) is being processedby the associated JM, at S152. If the output container is already beingprocessed by its associated or assigned JM, the job is placed back inthe queue and the job is marked by the QM as “pending”, at S154. If, onthe other hand, the output container is not being processed by itsassociated JM, the QM prepares instructions for a JM and launches the JMinstance and marks the job as “being processed”, at S156. The JMinstance then marks the job as completed, completed with warning orfailed, at S158. The QM can then initiate or start JMs for any newcontainers that may appear in a list of containers assigned to the QM.The QM can repeat the process starting at S150 by retrieving a nextpending job of the queue and marking the Job as “in-processing”, etc.

FIG. 26 is a diagram depicting a mathematical procedure used in readoperations to ensure that all read operations always begin at an SDBSaligned offset and are of a size that is also SDBS aligned, according toan embodiment of the present invention. When a container is created,every attempt is made to store headers and records such that everycomponent of each record and its headers are stored at SDBS alignedoffsets. However, there may be scenarios where a read operation may notoccur at SDBS offsets and the read request may not be of a size that isSDBS aligned. For example, one scenario may occur when the container iscopied from a storage device to another, and the SDBS of the sourcestorage device is smaller than the SDBS of the destination storagedevice. As a result, read operations may not occur at SDBS offsets.Another scenario occurs when the container record is in a 3^(rd) partyformat, and a viewport read or partial record read requires access tosmaller chunks of data spread across the record, the 3^(rd) party formatmay not have exercised the diligence of storing the data at offsets thatare SDBS aligned. For example, a JPEG2000 JP2 formatted image may storestiles which may not start at SDBS aligned offsets.

If the read operations begin at SDBS aligned offsets within a containerfile and if the size of a read operation is also SDBS aligned, the readoperation is much more efficient. To ensure SDBS aligned reads, the readoperation may be further optimized by reading a little bit extra numberof bytes.

For example, as depicted in FIG. 26, a read request may start fromoffset O of size S which may or may not be aligned to SDBS. If SDBS isdenoted by B_(sz), a new aligned read operation would start at offset O′and be of size S′ such that, O′=O−(O % B_(sz)) andS′=(|O−O′|+S)+(Bsz−((|O−O′|+S) % Bsz)) % Bsz). These equations providevalues to align a read operation to SDBS, if not already aligned.

As described in the above paragraphs, multiple large format images canbe stored in a single container. Information about the layout of theimage can be stored in the record metadata section. The image can bestored within the container (i) uncompressed, tiled, with the tilesarranged in column-major order, with optional overlapping, (ii)compressed, tiled, with the tiles arranged in row-major order, withoptional overlapping, (iii) compressed, tiled, with the tiles arrangedin column-major order, with optional overlapping, or (iv) stored in aknown 3^(rd) party file format that is tiled. Tile dimensions, startoffsets and sizes of each tile are stored in the record metadata sectionat the time the image is inserted or stored in the container.

In one embodiment, the image can be stored with overlapping in one ofthe three forms (i), (ii) and (iii). The fourth form (iv) involving3^(rd) party file formats is outside the scope of the concept ofoverlapped storage. In one embodiment, for overlapped tiles, the systemis configured to read only one tile of the overlapped tiles, and theread operation reads starting from a SDBS aligned location prior orequal to the start of that tile and ends at a SDBS aligned locationafter or equal to the end of the same tile. By reading from the SDBSaligned location prior or equal to the start of a desired tile to theSDBS aligned location after or equal to the end of the same tile, thesystem ensures that a tile as a whole is read.

FIG. 27A is a flow chart of a method for retrieval of an image viewportfrom a container, according to an embodiment of the present invention.The method for retrieving of viewport V ((X₀,Y₀), V_(w)×V_(h)) includesreading viewport V from an image I at record number N in container C, atS160. The method further includes validating the record number andviewport, at S162. In one embodiment, validating includes parsing therequest and verifying that the request is packaged correctly andverifying that the request is made for data that exists. If the requestis not packaged correctly or the request is not made for data that existthe request is invalidated and rejected. The method further includesreading record metadata section (e.g., at absolute container file offsetO_(rm) for record number N, where O_(rm)=S_(fcm)+(S_(r)×N), at S164,where S_(fcm) corresponds to the size of the file metadata section andS_(r) corresponds to the size of one complete record. The method theninquires whether the record is being updated, at S166. If the record isbeing updated, the method can retry or abort based on design policy(i.e., as implemented by the designer), at S168. If, on the other hand,the record is not being updated, the method determines if record Ncontains image I stored (i) uncompressed, tiled, column-major order,(ii) compressed, tiled, column-major order, (iii) compressed, tiled,row-major order, or (iv) in a third party file format. The method thenimplements procedure “A” depending on the above listed forms of storage,at S170 and presents the viewport for consumption (e.g., display on adisplay device), at S172.

FIG. 27B is a flow chart of part (procedure “A”) of the method forretrieval of an image viewport from a container, according to anembodiment of the present invention. The procedure “A” is in factdivided into four sub-procedures “A₁”, “A₂”, “A₃” and A₄” which areimplemented depending of the form of storage (i), (ii), (iii) or (iv).For example, procedure “A₁” is implemented for an image I which isstored uncompressed, tiled, with the tiles arranged in column-majororder, with optional overlapping, in record N. The sub-procedure “A₁”includes computing a top left and bottom right column numbers of thetiles containing the viewport, at S180. For example, T(C₀) is the lefttile column, T(C₁) is the right tile column and there are a total ofC_(i) columns. The sub-procedure A₁ further includes, for each columnC_(k) from C₀ to C₁, computing the absolute aligned offset from thestart of the container file, and the aligned size of the strip startingfrom line Y₀ to and including line Y₀+V_(h)−1 as described in the aboveparagraphs (see, for example, FIGS. 10A and 10B and relateddescription), and dispatching parallel or asynchronous read requests ofcomputed sizes, at S182. The sub-procedure A₁ further includes as eachparallel or asynchronous read request completes, collecting andprocessing only pixels that reside within the viewport bounds togenerate the output viewport in a single raster, at S184.

Sub-procedure “A₂” is implemented for an image I which is storedcompressed, tiled, with the tiles arranged in row-major order, withoptional overlapping, in record N. The sub-procedure “A₂” includescomputing a top left and bottom right column and row numbers of thetiles containing the viewport, at S190. For example, T(C₀,R₀) is thetop-left tile, T(C₁,R₁) is the bottom-right tile and there are a totalof C_(i) columns and R_(j) rows. The sub-procedure “A₂” furtherincludes, for each row R_(k) from R₀ to R₁, launching parallel orasynchronous operations to read tiles for each row, at S192. Each rowR_(k) contains C_(i) tiles from tile C₀ to C₁. These tiles are stored insequence within the container and can be regarded as a tile strip. Thesub-procedure “A₂” includes reading the entire tile strip for the row ina single read operation or reading one or more individual tiles inparallel or in asynchronous operation, at S194. If a tile strip is read,the read starts from a SDBS aligned location prior to or equal to thestart of tile C₀, R_(k) and ends at a SDBS aligned location after orequal to the end of the tile C₁, R_(k). If the sub-procedure “A₂” readsmultiple individual tiles in parallel or asynchronously, each tile readstarts from a SDBS aligned location prior to or equal to the start ofthat tile C_(m), R_(k) and ends at a SDBS aligned location after orequal to the end of the same tile C_(m), R_(k), where C_(m) ranges fromC₀ to C₁. As each parallel or asynchronous read request completes, thesub-procedure “A₂” decodes, collects and processes only pixels thatreside within the viewport bounds to generate the output viewport in asingle raster, at S196.

Sub-procedure “A₃” is implemented for an image I which is storedcompressed, tiled, with the tiles arranged in column-major order, withoptional overlapping, in record N. The sub-procedure “A₃” includescomputing a top left and bottom right column and row numbers of thetiles containing the viewport, at S200. For example, T(C₀,R₀) is thetop-left tile, T(C₁,R₁) is the bottom-right tile and there are a totalof C₁ columns and R_(j) rows. The sub-procedure “A₃” further includes,for each column C_(k) from C₀ to C₁, launching parallel or asynchronousoperations to read tiles for each column, at S202. Each column C_(k)contains R_(j) tiles from tile R₀ to R₁. These tiles are stored insequence within the container and can be regarded as a tile strip. Thesub-procedure “A₃” includes reading the entire tile strip for the columnin a single read operation or reading one or more individual tiles inparallel or in asynchronous operation, at S204. If a tile strip is read,the read starts from a SDBS aligned location prior to or equal to thestart of tile C_(k), R₀ and ends at a SDBS aligned location after orequal to the end of the tile C_(k), R₁. If the sub-procedure “A₃” readsmultiple individual tiles in parallel or asynchronously, each tile readstarts from a SDBS aligned location prior to or equal to the start ofthat tile C_(k), R_(m) and ends at a SDBS aligned location after orequal to the end of the same tile C_(k), R_(m), where R_(m) ranges fromR₀ to R₁. As each parallel or asynchronous read request completes, thesub-procedure “A₃” decodes, collects and processes only pixels thatreside within the viewport bounds to generate the output viewport in asingle raster, at S206.

Sub-procedure “A₄” is implemented for an image I which is stored in athird party tiled file format. The sub-procedure “A₄” includes computinga top left and bottom right column and row numbers of the tilescontaining the viewport, at S210. For example, T(C₀,R₀) is the top-lefttile, T(C₁,R₁) is the bottom-right tile and there are a total of C_(i)columns and R_(j) rows. The sub-procedure “A₄” includes determiningwhether the record metadata section holds a table of offsets and sizesof each tile in the image that exists in a known third-party file format(e.g., JPEG2000 or TIFF). The sub-procedure “A₄” further includesinquiring whether the image data is stored in row-major order or columnmajor order, at S214. If the image data is stored in column-major order,sub-procedure “A₃” is used instead and instead of computing tileoffsets, the tile offsets are determined from the record metadatasection, at S216.

If the image data is stored in row-major order, sub-procedure “A₂” isused instead, and instead of computing tile offsets, the tile offsetsare determined from the record metadata section, at S218. If the orderis not known, either sub-procedure “A₂” or sub-procedure “A₃” can beemployed to read multiple individual tiles and instead of computingoffsets the offsets can be retrieved from the record metadata section,at S220. As each parallel or asynchronous read request completes, thesub-procedure “A₄” decodes, collects and processes only pixels thatreside within the viewport bounds to generate the output viewport in asingle raster, at S222.

FIG. 28 is a schematic representation of a viewport within an imagestored within a container file, according to an embodiment of thepresent invention. For example, a viewport V starting from pixel X₀,Y₀and having a size V_(w)×V_(h) pixels within a given image I that is at arecord numbered N in a given container is retrieved using a method forretrieving a viewport, according to an embodiment of the presentinvention. A number M of such viewports may be read from the samecontainer storing a plurality of images using the same method, appliedeither in sequence or in parallel using a local or distributed computingsystem. In one embodiment, the origin X₀,Y₀ of each of the plurality ofviewports within the plurality of images is fixed from one viewport to anext viewport. In another embodiment, the origin X₀,Y₀ of each of theplurality of viewports is movable from one viewport to a next viewport.

In one embodiment, a viewport V can be requested for display on adisplay device. Alternatively or in addition, a viewport may also berequested for further processing by a processing device to analyze datain the viewport or extract data from the viewport. For example, theprocessor may perform image-processing operations on the content of manysuch viewports (e.g., perform processing on a sequence of viewportsobtained during a certain time period) in order to detect differencesbetween image content or data in the viewports that would indicate, forexample, activity-based information. For example, by processingviewports from large format images that were captured by placing sensorson an aircraft pointing towards the ground, and persistently hoveringover a fixed location on the ground, processing operations on viewportscan generate tracks for moving objects. Efficient retrieval anddissemination of viewports for processing by processors may thus providean efficient means for retrieval of desired information. The retrieveddesired information may be communicated to a user in various forms suchas text, graphics, audio signal, etc. The retrieved desired informationmay even be further displayed on a display device in a form of an imageor a series of images (e.g., a movie). Therefore, as it can beappreciated from the above paragraphs, the method and system describedherein can be used in dissemination of viewports for further processingby a processing device and/or further display by a display devicedepending on the application sought.

FIG. 29 is a flow chart of a method for retrieval of an image viewportfrom a container, according to an embodiment of the present invention.In one embodiment, the method can be implemented as a web service thatuses the HTTP protocol. However, the method can also be implementedusing other protocols. In one embodiment, a basic request will performone of two tasks. The request may ask for a viewport from a singlerecord within a container. This is referred to herein as a“single-record” request. Alternatively, the request may ask for aviewport from a sequence of multiple records within a container, wherethe viewport may remain static or may follow a preset temporal andspatial path. This is referred to herein as a “multi-record” request.

A single-record request is efficient for reading different viewports fornon-sequential records or when displaying data in the form of aflip-book. Such a request is also efficient when the nature of the nextrequest is random and cannot be anticipated. A multi-record request isefficient for reading a viewport from a group of sequential records. Oneexample of application of a multi-record request is when requesting asequence of viewports from a container for eventual encoding into anMPEG codestream. In another example, if the network latency is high,many single-record requests translate into many individual requests, andwill take longer to execute because of network latency overhead perrequest. However, a multi-record request is a single web request, and itretrieves several viewports within that one request, thus minimizing thenetwork latency overhead.

A single-record request is denoted by P. A single record requestinvolves reading a viewport V_(i) from a record N_(j) from a containerC. Viewport V_(i) starts from top left pixel (X₀,Y₀) and is V_(w)×V_(h)pixels in dimensions. The result is delivered to the client in a knownformat F, where F can be raw pixel data or may be converted into animage of known format, for example JPEG, JPEG2000, TIFF, or PNG.Optionally, metadata regarding the viewport may also be packaged intothe result using a standard HTTP reply format, for example a multi-partmessage that follows IETF RFC2045.

The request P is denoted by P(C, N_(j), V_(i)((X₀,Y₀),V_(w)×V_(h)), F).Additional input parameters may be packaged as part of the request asneeded for the service implementation.

The method of retrieving or reading the viewport includes inputting therequest and reading the parameters of the request P includingidentifying the position and size of the viewport V_(i)(X₀,Y₀,V_(w)×V_(h)), the record containing the viewport V_(i) and thecontainer containing the record N_(j), at S230. The method furtherincludes opening the container C, at S232. The opening may be anexplicit opening or a retrieving from a cached container. The methodfurther includes reading the viewport V_(i) (X₀,Y₀,V_(w)×V_(h)) fromrecord N_(j) from container C (see, for example, FIG. 28), at S234. Themethod includes transferring the result into format F and transcode orencode the result if needed, at S236. The method further includesoptionally closing the container, at S238, and outputting or presentingor streaming the resulting data to the user, at S240. At any pointduring opening of the container, reading of the viewport, and/ortransferring the result into format F, the method may generate an errormessage and prepare a reply to handle error, at S242.

A multi-record request is denoted by L. L has more than one P. L isdenoted by L(P₁ . . . P_(n)) and is read as “L is a multi-record requestcontaining multiple single record requests from P₁ to P_(n),” where eachP_(i) is a single-record request. From the user's perspective L istreated as a single operation. From the service (e.g., web service)implementation's perspective, L performs multiple P operations. Each Pis independent of another P. Therefore, each single record requestsP_(i) can be executed in parallel. The result of any P_(i) is deliveredin the format F_(i) chosen for that P_(i).

The user makes one L request. The service processing an L request, onthe other hand, makes multiple P_(i) requests in parallel. Each requestP_(i) may be completed out of order by the service implementation. Eachrequest P_(i) can be distributed across other systems or launched to runlocal to a computer system.

The user may either want to receive the result of each P request in thesame order as it was sent, from P₁ to P_(n), or the client may want toreceive the results of all P requests as soon as they complete, whichmay be out of order. In the latter case, the user may further order theresults if desired.

FIG. 30A is a flow chart depicting a method for retrieval of multipleimage viewports from a container, the method being implemented as a webservice, according to an embodiment of the present invention. The methodincludes receiving a request L (P₁ . . . P_(n)) from a user, at S250,where P₁ . . . P_(n) are single record requests. The method furtherincludes validating parameters of the request L, at S252. In oneembodiment, validating parameters of the request includes validatingeach request P₁ . . . P_(n) and thus validating parameters of eachrequest P (i.e., validating parameters such as container, recordlocation, etc of each request P). In one embodiment, the method furtherincludes dispatching n requests from P₁ . . . P_(n) in parallel, at S254and wait for n requests to complete, at S256 and ending, at S258.

FIG. 30B is a flow chart depicting a sub-procedure within the methodshown in FIG. 30A. In one embodiment, as shown in FIG. 30B, thedispatching of ‘n’ requests from P₁ . . . P_(n) in parallel(“dispatching procedure”) includes, for each request P_(i) from P₁ . . .P_(n) running in parallel, sending request P_(i) to a single recordviewport request processing service (see, FIG. 29 and relateddescription), at S260. The parallel requests P₁ . . . P_(n) can run overa distributed network of computer or locally within one computer. On anetwork with high latency, it may be beneficial to run the parallelrequests P₁ . . . P_(n) locally. The dispatching further includesreceiving results of each request P_(i), at S262. The dispatchingfurther includes inquiring whether preserving request order is desired,at S264. In one embodiment, if preserving the order of the request isdesired, the dispatching collects the result of each P_(i) request, atS266, and send the result of each P_(i) in order from P₁ to P_(n) as theresult becomes available, at S268. In one embodiment, if preserving theorder of the request is not desired, the dispatching sends the result ofeach P_(i) as soon as the result is available, at S267. The dispatchingprocedure then ends, at S269.

A very large format motion imagery container C contains very largeindividual frames or images. One such large image or frame generallydoes not fit on a typical high-resolution display. Generally, a userrequests only a part of the image which is a viewport image. If a userdesires to flip through a plurality of viewports in sequence, this canbe achieved by making multiple requests to a single-record or amulti-record viewport request processing service. However, this may notbe the most efficient way in terms of utilizing a network bandwidthbetween a server computer or computers and a client computer.

An alternative way to provide the sequence of viewports (SOV) for aclient that desires to play the sequence of viewports is to encode thesequence of viewports into a single video codestream of a known format,such as, but not limited to, MPEG, AVI, MPEG-2, MKV, or WMV. It isassumed that such a video codestream comprises several sub-components,such that each sub-component of a specific type can be encodedindependently from other sub-components of that type. For example, invideo coding, a group of pictures (GOP) specifies the order in whichindividual frames are arranged. For example, a GOP can contain one ormore of an intra-frame (I-frame), a predicted-frame (P-frame), abidirectional-frame (B-frame), or a DC-picture (D-frame). A GOP maycontain additional frame types as well. One or more sequential GOPs in avideo codestream can be regarded as a sub-component.

A video codestream comprises multiple independent sub-componentcodestreams. Each sub-component codestream includes one or more verylarge format motion imagery viewports as input. An example is MPEG-2that wraps an H.264 video codestream where one or more H.264 encodedGOPs comprise a video sub-component. FIG. 31A is a diagram depicting theinternal breakdown of a video sub-component comprising a videocodestream of one or more group of pictures (GOP), according to anembodiment of the present invention.

A video sub-component requires a sequence of viewports as source datafrom a container. A service implementation requests the viewports andencodes the viewports into a codestream for the video sub-component.

In one embodiment, a method of retrieving a sequence of viewports from acontainer as a web service for the purpose of encoding the viewportsinto a video sub-component is provided. FIG. 31B is a diagram depictinga data path of a method for providing a streaming video sub-component asa web service derived from multiple image viewports from a container,according to an embodiment of the present invention. In one embodiment,a user issues a request for streaming video of a sequence of viewports,denoted by G. The request G comprises a single request L with the sameformat for each viewport request P within the request L. Thus, request Gcan be denoted by G(L, F), where F is the format of the output picture.The service implementation may choose to send an L request to anotherservice or implement L as part of the implementation of G. Viewportrequests P₁ . . . P_(n) corresponding to sequence of viewports V can bestreamed sequentially, each in a format F. The video sub-componentencoding service then encodes the retrieved viewports into a videosubcomponent codestream and send the codestream to the client or user(see FIG. 31B).

FIG. 32 is a flow chart depicting a method for providing a streamingvideo service of a group of pictures (GOP) from a container, accordingto an embodiment of the present invention. The method includes receivinga request for streaming video of a sequence of viewports, denoted by G(L,F), at S270. The request G comprises a single request L with the sameformat for each viewport request P_(i) within the request L. Eachviewport request P_(i) corresponds to requested viewport V_(i). Themethod further includes validating the request G, at S272. In oneembodiment, the validating includes checking parameters of the request Gincluding checking if the viewports requested exist within specifiedrecords within a specified container. The method further includessending the request L of the request G, at S274. In one embodiment, therequest is made with the state of the “preserve request order” flag setto “true”. The method further includes encoding each received viewportrequest P_(i) into a video codestream subcomponent, at S276, and streamencoded codestream in expected sequence, at S278 and end the method atS280. During the validation of the request, the sending of the request Lthe encoding of picture P_(i), an error may occur. In which case, themethod may further include processing the error into a reply and send tothe user, at S279.

A request for streaming video is denoted by D. The request for streamingvideo D contains one or more parts. Each part originates from aspecified container. Each part contains video frames. Each video frameis a viewport from a large format motion imagery frame. In oneembodiment, a method of retrieving a video codestream of a collection ofa sequence of viewports from one or more containers is provided. Eachsequence of viewports is retrieved and encoded by a video streamingservice that provides a partial video codestream as described in theabove paragraphs and each such partial video codestream is processed andstreamed by a video streaming service. Such a video streaming service isdenoted herein by M. The video streaming service M processes a request Dwhich is a collection of requests for a sequence of viewports from oneor more containers. M is a video streaming service. D is the requestthat M serves. Consider that the video streaming request D comprises ofa total of Q requests from G₁ to G_(Q) where each G_(i) is a request fora sequence of viewports from a container. D can be denoted by D(G₁ . . .G_(Q), F_(D)) where F_(D) is the format in which the final videocodestream is streamed to the client. Each request G_(i)(L, F) isprovided with a format F selected by D such that the transformation fromF to F_(D) can be further optimized.

When the implementation of the web service receives such a request D, itcan execute each request G_(i) in parallel. As video streaming service Mreceives the video sub-component codestream from each G_(i), M packagesit for transmission. This can also be performed in parallel, which meansthat G₁, G₂, . . . , G_(Q) can be launched in parallel. The results mayarrive into M (i.e., into the video server) out of order. They areassembled by M to be in the correct order. The resulting final videocodestream is streamed to the client for consumption. Thus, a streamingservice M serves a video request D.

As shown in FIG. 33, the video codestream comprises a first series ofgroup of pictures GOP₁, GOP₂, . . . , GOP_(N). Between each GOP, (iranging from 1 to N) may be interleaved with additional data such as,for example, sound or audio information, close captioned data, or KeyLength Value (KLV) metadata or other reference data, or any combinationthereof. For example, GOP₁, GOP₂, . . . , GOP_(N) may correspond to aresult of request G₁(L,F) processed by streaming service M to deliverthe completed video codestream. For example, the video codestream mayfurther include a second series of group of pictures GOP₁, GOP₂, . . . ,GOP_(N) which may correspond to a result of a next request G₂(L,F)processed by M to deliver the complete video codestream, etc. As shownin FIG. 33, the requests G₁, G₂, . . . , G_(Q) may be processed inparallel by the implementation of M.

FIG. 34A is a flow chart depicting a method of generating a completevideo codestream using multiple parallel video sub-component requests,according to an embodiment of the present invention. The method includesreceiving a request D (G₁ . . . G_(Q), F_(D)) from a user, at S290,where G₁ . . . G_(Q) are requests for series of group of pictures(GOPs). The method further includes validating parameters of the requestD, at S292. In one embodiment, validating parameters of the requestincludes validating each request G₁ . . . G_(Q). In one embodiment, themethod further includes dispatching Q requests from G₁ . . . G_(Q) inparallel, at S294 and wait for Q requests to complete, at S296 andending, at S298.

FIG. 34B is a flow chart depicting a sub-procedure within the methodshown in FIG. 34A. In one embodiment, as shown in FIG. 34B, thedispatching of Q requests from G₁ . . . G_(Q) in parallel includes, foreach request G_(i) from G₁ . . . G_(Q) running in parallel, sendingrequest G_(i) to a video sub-component request processing service (see,FIG. 33 and related description), at S300. The parallel requests G₁ . .. G_(Q) can run over a distributed network of computer or locally withinone computer. On a network with high latency, it may be beneficial torun the parallel requests G₁ . . . G_(Q) locally. The dispatchingfurther includes receiving results of each request G_(i), at S302. Thedispatching further includes collecting and processing the result fromthe request G_(i), at S304. In one embodiment, the dispatching mayinclude performing any interleaving of data between GOPs if desired. Thedispatching further includes streaming G_(i)'s result in order from G₁to G_(Q) as it becomes available, at S306. The dispatching sub-procedurethen ends, at S308.

The result is a streaming video service M that effectively encodes videofrom viewports that were extracted from large format motion imagerystored in containers. The system is independent of the codec in whichthe video clip is desired, so long as there is a logical ability for thecodec and the video format to encode data as a sequence of videosub-components as described above.

In some embodiments, programs for performing methods in accordance withembodiments of the invention can be embodied as program products in acomputer such as a personal computer or server or in a distributedcomputing environment comprising a plurality of computers. The computermay include, for example, a desktop computer, a laptop computer, ahandheld computing device such as a PDA, a tablet, etc. The computerprogram products may include a computer readable medium or storagemedium or media having instructions stored thereon used to program acomputer to perform the methods described above. Examples of suitablestorage medium or media include any type of disk including floppy disks,optical disks, DVDs, CD ROMs, magnetic optical disks, RAMs, EPROMs,EEPROMs, magnetic or optical cards, hard disk, flash card (e.g., a USBflash card), PCMCIA memory card, smart card, or other media.Alternatively, a portion or the whole computer program product can bedownloaded from a remote computer or server via a network such as theinternet, an ATM network, a wide area network (WAN) or a local areanetwork.

Stored on one or more of the computer readable media, the program mayinclude software for controlling both the hardware of a general purposeor specialized computer or processor. The software also enables thecomputer or processor to interact with a user via output devices such asa graphical user interface, head mounted display (HMD), etc. Thesoftware may also include, but is not limited to, device drivers,operating systems and user applications.

Alternatively, instead or in addition to implementing the methodsdescribed above as computer program product(s) (e.g., as softwareproducts) embodied in a computer, the method described above can beimplemented as hardware in which for example an application specificintegrated circuit (ASIC) or graphics processing unit or units (GPU) canbe designed to implement the method or methods of the present invention.

Although the various steps of the above method(s) are described in theabove paragraphs as occurring in a certain order, the presentapplication is not bound by the order in which the various steps occur.In fact, in alternative embodiments, the various steps can be executedin an order different from the order described above.

Although the invention has been described in detail for the purpose ofillustration based on what is currently considered to be the mostpractical and preferred embodiments, it is to be understood that suchdetail is solely for that purpose and that the invention is not limitedto the disclosed embodiments, but, on the contrary, is intended to covermodifications and equivalent arrangements that are within the spirit andscope of the appended claims. For example, it is to be understood thatthe present invention contemplates that, to the extent possible, one ormore features of any embodiment can be combined with one or morefeatures of any other embodiment.

Furthermore, since numerous modifications and changes will readily occurto those of skill in the art, it is not desired to limit the inventionto the exact construction and operation described herein. Accordingly,all suitable modifications and equivalents should be considered asfalling within the spirit and scope of the invention.

What is claimed:
 1. A computer system for processing a viewport withinan image arranged as a matrix of tiles from a container file, thecomputer system comprising: one or more processors configured to:receive a request to read data of the viewport starting from a point oforigin of the viewport, the viewport having a pixel width and a pixelheight, the viewport being a portion of an image that is stored in arecord within the file container; compute column numbers or row numbers,or both, of the tiles containing the viewport; and read in parallel orasynchronously each row or each column of tiles containing the viewport.2. The computer system according to claim 1, wherein the one or moreprocessors are configured to reading record metadata of the recordcontaining the image that includes the viewport, the record metadataincludes a record metadata section at absolute container file offsetO_(rm), where O_(rm)=S_(fcm)+(S_(r)×N), where S_(fcm) corresponds to asize of the record metadata section and S_(r) corresponds to a size ofthe record, and N is a record number.
 3. The computer system accordingto claim 1, wherein the one or more processors are configured todetermine whether the image containing the viewport is stored:uncompressed, tiled arranged in a column major order, or compressed,tiled and arranged in column major order, or compressed, tiled andarranged in a row-major order, or in a third party file format.
 4. Thecomputer system according to claim 3, wherein if the image isuncompressed, tiled arranged in a column major order, the one or moreprocessors are configured to: compute top left and bottom right columnnumbers of the tiles containing the viewport, the viewport having a topleft corner at X₀,Y₀, and having a pixel width V_(w) and pixel heightV_(h), for each column between the top left column number and the bottomright column number, compute an absolute aligned offset from a start ofthe container file, and an aligned size of a strip starting from line Y₀to and including line Y₀+V_(h)−1, and collect and process only pixelsthat reside within bounds of the viewport to generate the outputviewport in a single raster.
 5. The computer system according to claim3, wherein if the record contains an image compressed, tiled, with thetiles arranged in row-major order, the one or more processors areconfigured to: compute top left and bottom right column and row numbersof the tiles containing the viewport, the viewport having a top leftcorner at X₀,Y₀, and having a pixel width V_(w) and pixel height V_(h),for each row between the top left row number and the bottom right rownumber, launch parallel or asynchronous operations to read tiles foreach row.
 6. The computer system according to claim 5, wherein the oneor more processors are configured to read an entire tile strip for eachrow in a single read operation or reading one or more individual tilesin parallel or in asynchronous operation.
 7. The computer systemaccording to claim 6, wherein the one or more processors are configuredto read starting from a storage device block size (SDBS) alignedlocation prior to or equal to a start of an initial tile in the tilestrip and ending at a storage device block size (SDBS) aligned locationafter or equal to an end of a last tile in the tile strip.
 8. Thecomputer system according to claim 6, wherein the one or more processorsare configured to read multiple individual tiles in parallel orasynchronously, wherein each tile read starts from a storage deviceblock size (SDBS) aligned location prior to or equal to the start of atile and ends at a storage device block size (SDBS) aligned locationafter or equal to the end of the same tile.
 9. The computer systemaccording to claim 8, wherein the one or more processors are configuredto collect and process only pixels that reside within viewport bounds togenerate an output viewport in a single raster.
 10. The computer systemaccording to claim 3, wherein if the record contains an image storedcompressed, tiled, with the tiles arranged in column-major order, theone or more processors are configured to: compute top left and bottomright column and row numbers of the tiles containing the viewport, theviewport having a top left corner at X₀,Y₀, and having a pixel widthV_(w) and pixel height V_(h), and for each column, launch parallel orasynchronous operations to read tiles for each column between the topleft column number and the bottom right column number.
 11. The computersystem according to claim 10, wherein the one or more processors areconfigured to read an entire tile strip for the column in a single readoperation or read one or more individual tiles in parallel or inasynchronous operation.
 12. The computer system according to claim 11,wherein the one or more processors are configured to read starting froma storage device block size (SDBS) aligned location prior to or equal toa start of an initial tile and ending at a storage device block size(SDBS) aligned location after or equal to an end of a last tile.
 13. Thecomputer system according to claim 11, wherein the one or moreprocessors are configured to read multiple individual tiles in parallelor asynchronously, wherein each tile read starts from a storage deviceblock size (SDBS) aligned location prior to or equal to a start of atile and ends at a storage device block size (SDBS) aligned locationafter or equal to the end of the same tile.
 14. The computer systemaccording to claim 13, wherein the one or more processors are configuredto collect and process only pixels that reside within the viewportbounds to generate the output viewport in a single raster.
 15. Thecomputer system according to claim 1, wherein if the record contains animage stored in a third party tiled file format, the one or moreprocessors are configured to: compute a top left and bottom right columnand row numbers of the tiles containing the viewport, the viewporthaving a top left corner at X₀,Y₀, and having a pixel width V_(w) andpixel height V_(h), determine that record metadata of the recordcontaining the image that includes the viewport holds a table of offsetsand sizes of each tile in the image that exists in a known third-partyfile format, and determine whether data in the image is stored inrow-major order or column-major order.
 16. The computer system accordingto claim 15, wherein if the image data is stored in the column-majororder, the one or more processors are configured to: compute a top leftand bottom right column and row numbers of the tiles containing theviewport, for each column between the top left column number and thebottom right column number, launch parallel or asynchronous operationsto read tiles for each column, and read an entire tile strip for eachcolumn in a single read operation or reading one or more individualtiles in parallel or in asynchronous operation.
 17. The computer systemaccording to claim 15, if the image data is stored in row-major order,the one or more processors are configured to: compute a top left andbottom right column and row numbers of the tiles containing theviewport, for each row between the top left row number and the bottomright row number, launch parallel or asynchronous operations to readtiles for each row, and read the entire tile strip for each row in asingle read operation or reading one or more individual tiles inparallel or in asynchronous operation.
 18. The computer system accordingto claim 1, wherein the one or more processors are configured toreceiving a hypertext transfer protocol (HTTP) request.
 19. The computersystem according to claim 1, wherein the one or more processors areconfigured to output the data of the viewport.
 20. The computer systemaccording to claim 19, wherein the one or more processors are configuredto transmit the data of the viewport using hypertext transfer protocol(HTTP) or display the data of the viewport on a display device.
 21. Acomputer system for processing a plurality of viewports in a pluralityof images, each viewport being wholly contained within an image in theplurality of images, each image in the plurality of images beingarranged as a matrix of tiles, the plurality of images being stored in acontainer file, the computer system comprising: one or more processorsconfigured to: receive a request to read data of each of the pluralityof viewports starting from a point of origin of each of the plurality ofviewports, each viewport having a pixel width and a pixel height, andeach viewport being a portion of an image in the plurality of images,each image is stored in a record in a plurality of records within thefile container; compute column numbers or row numbers, or both, of thetiles containing each viewport; and read in parallel or asynchronouslyeach row or each column of tiles containing each viewport.
 22. Thecomputer system according to claim 21, wherein the one or moreprocessors are configured to output each of the plurality of viewports.23. The computer system according to claim 21, wherein the point oforigin of each of the plurality of viewports is fixed from one viewportcontained in one image to another viewport contained in another image.24. The computer system according to claim 21, wherein the point oforigin of each of the plurality of viewports is different from oneviewport to another viewport.
 25. The computer system according to claim21, wherein the pixel width and the pixel height of each viewport in theplurality of viewports is the same for all the viewports.
 26. Thecomputer system according to claim 21, wherein the one or moreprocessors are configured to read record metadata of each recordcontaining a corresponding image that contains each viewport.
 27. Amethod of processing a plurality of viewports in a plurality of images,each viewport being wholly contained within an image in the plurality ofimages, each image in the plurality of images being arranged as a matrixof tiles, the plurality of images being stored in a container file, themethod being implemented by a computer system that includes one or moreprocessors configured to execute processing operations, the methodcomprising: computing, by the one or more processors, column numbers orrow numbers, or both, of tiles containing each viewport, starting from apoint of origin of each of the plurality of viewports, the point oforigin of each of the plurality of viewports is fixed or different fromone viewport contained in one image to another viewport contained inanother image, each viewport having a pixel width and a pixel height,and each viewport being a portion of an image in the plurality ofimages, each image is stored in a record in a plurality of recordswithin the file container; and reading, by the one or more processors,in parallel or asynchronously each row or each column of tilescontaining each viewport.
 28. The method according to claim 27, furthercomprising outputting each of the plurality of viewports.
 29. The methodaccording to claim 27, wherein the pixel width and the pixel height ofeach viewport in the plurality of viewports is the same for all theviewports.
 30. The method according to claim 27, further comprisingreading, by the one or more processors, record metadata of each recordcontaining a corresponding image that contains each viewport.
 31. Themethod according to claim 27, further comprising reading an entire tilestrip for each row or for each column in a single read operation orreading one or more individual tiles in parallel or in asynchronousoperation.
 32. The method according to claim 31, wherein reading thetile strip in parallel or in asynchronous operation comprises readingstarting from a storage device block size (SDBS) aligned location priorto or equal to a start of an initial tile in the tile strip and endingat a storage device block size (SDBS) aligned location after or equal toan end of a last tile in the tile strip.
 33. The method according toclaim 31, further comprising reading multiple individual tiles inparallel or asynchronously, wherein each tile read starts from a storagedevice block size (SDBS) aligned location prior to or equal to the startof a tile and ends at a storage device block size (SDBS) alignedlocation after or equal to the end of the same tile.
 34. A method ofprocessing a plurality of viewports in a plurality of images, eachviewport being wholly contained within an image in the plurality ofimages, each image in the plurality of images being arranged as a matrixof tiles, the plurality of images being stored in a container file, themethod being implemented by a computer system that includes one or moreprocessors configured to execute processing operations, the methodcomprising: receiving, by the one or more processors, a request to readdata of a first viewport starting from a first point of origin of thefirst viewport and to read a second viewport starting from a secondpoint of origin of the second viewport, the first viewport having afirst pixel width and a first pixel height and the second viewporthaving a second pixel width and a second pixel height, the firstviewport being a portion of a first image in the plurality of images andcontaining a plurality of tiles and the second viewport being a portionof a second image in the plurality of images and containing a pluralityof tiles, the first image is stored in a first record within the filecontainer and the second image is stored in a second record within thefile container; computing, by the one or more processors, column numbersor row numbers, or both, of tiles containing the first viewport;computing, by the one or more processors, column numbers or row numbers,or both, of tiles containing the second viewport; reading, by the one ormore processors, in parallel or asynchronously each row or each columnof tiles containing the first viewport; and reading, by the one or moreprocessors, in parallel or asynchronously each row or each column oftiles containing the second viewport.
 35. The method according to claim34, further comprising outputting the first viewport and the secondviewport in sequence.
 36. The method according to claim 35, whereinoutputting the first viewport and the second viewport comprisesdisplaying the first viewport and the second viewport in sequence on adisplay device.
 37. The method according to claim 34, wherein the firstpoint of origin and the second point of origin are the same.
 38. Themethod according to claim 34, wherein the first point of origin and thesecond point of origin are different.
 39. The method according to claim34, wherein the first pixel width is the same as the second pixel widthand the first pixel height is the same as the second pixel height. 40.The method according to claim 34, further comprising reading, by the oneor more processors, record metadata of the first record and metadata ofthe second record.
 41. The method according to claim 34, wherein readingeach row or each column of tiles containing the first viewport comprisesreading each row or each column of tiles starting from a storage deviceblock size (SDBS) aligned location prior to or equal to a start of aninitial tile in the plurality of tiles and ending at a storage deviceblock size (SDBS) aligned location after or equal to an end of a lasttile in the plurality of tiles.
 42. The method according to claim 34,wherein reading each row or each column of tiles containing the secondviewport comprises reading each row or each column of tiles startingfrom a storage device block size (SDBS) aligned location prior to orequal to a start of an initial tile in the plurality of tiles and endingat a storage device block size (SDBS) aligned location after or equal toan end of a last tile in the plurality of tiles.